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Human-Computer Interface Including Haptically Controlled Interactions 

This invention was made with Government support under Contract DE-AC04-94AL85000 
awarded by the U. S. Department of Energy. The Government has certain rights in the 



invention. 



Priority Claim 



This application claims the benefit of U.S. Provisional Application 60/202,448, filed on 
05/06/2000, and U.S. Patent Application 08/834,642, filed on 4/14/1997, both incorporated 



herein by reference. 



Background of the Invention 



[0001] This invention relates to the field of human-computer interfaces, specifically those relating 
to methods of intuitively controlling interactions such as scrolling and zooming. 
[0002] Computing technology has seen a many-fold increase in capability in recent years. 
Processors work at ever higher rates; memories are ever larger and faster; mass storage is larger 
5 and cheaper every year. Computers now are essential elements in many aspects of life, and are 
often used to present three dimensional worlds to users, in everything from games to scientific 
visualization. 

[0003] The interface between the user and the computer has not seen the same rate of change. 
Screen windows, keyboard, monitor, and mouse are the standard, and have seen little change 
10 since their introduction. Many computers are purchased with great study as to processor speed, 
memory size, and disk space. Often, little thought is given to the human-computer interface, 
although most of the user's experience with the computer will be dominated by the interface (rarely 
does a user spend significant time waiting for a computer to calculate, while every interaction must 
use the human-computer interface). 
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Present human-computer interfaces generally rely on visual feedback. A document in a word 
processor or editor, for example, is displayed to the user. The user can position a cursor by 
moving a mouse; visual feedback of the display of a cursor allows the user to effectively control 
the cursor position. The user can also edit the document by typing, cutting, or pasting, for 

5 example, again using visual feedback to effectively control the computer application. 

[0004] Using visual feedback for all aspects of a human-computer interface, however, can require 
that the user constantly change the visual focus. For example, in a word processor, the user can 
focus on the text, directing visual attention to the text and the edits the user is making. Movement 
of the cursor, or scrolling through the document, requires that the user divert attention from the 

10 text editing and to the cursor control or scrolling operation. Once the cursor has been repositioned 
or the document scrolled to the appropriate place, then the user must once again redirect attention 
from the scrolling and cursor interaction to the editing interaction. The overload of the visual 
feedback path (by sharing it for various different functions) can result in decreased human- 
computer interaction efficiency. Further, fine control of scrolling can be difficult using just visual 

15 feedback, a difficulty made even worse by the overload of the visual feedback path. 

[0005] Accordingly, there is a need for new methods of human-computer interaction that make use 
of other feedback paths, reducing the overload of the visual feedback path. 

Summary of the Invention 

[0006] The present invention provides a method of human-computer interfacing that provides 
20 haptic feedback to control interface interactions such as scrolling or zooming within an application. 
Haptic feedback in the present method allows the user more intuitive control of the interface 
interactions, and allows the user's visual focus to remain on the application. The method 
comprises providing a control domain within which the user can control functions such as scrolling 
or zooming. For example, a haptic boundary can be provided corresponding to scrollable or 
25 scalable portions of the application domain. The user can position a cursor near such a boundary, 
feeling its presence haptically (reducing the requirement for visual attention for control of scrolling 
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of the display). The user can then apply force relative to the boundary, causing the interface to 
scroll the domain. The rate of scrolling can be-related to the magnitude of applied force, providing 
the user with additional intuitive, non-visual control of scrolling. 

[0007] The method also comprises providing a control domain, corresponding to scrollable or 
5 scalable directions in the application domain. The user can control scrolling or zooming in the 
application domain by interacting with the control domain as discussed previously for the 
application domain scrollable boundaries. 

[0008] Advantages and novel features will become apparent to those skilled in the art upon 
examination of the following description or may be learned by practice of the invention. The 
10 objects and advantages of the invention may be realized and attained by means of the 
instrumentalities and combinations particularly pointed out in the appended claims. 

Description of the Figures 

[0009] The accompanying drawings, which are incorporated into and form part of the specification, 
illustrate embodiments of the invention and, together with the description, serve to explain the 
15 principles of the invention. 

[0010] Figure 1 is an illustration of a computer display, illustrating display of a portion of an item 
and associated visual and haptic boundaries. 

[001 1] Figure 2 is an illustration of a computer display and a control zone. 
[0012] Figure 3a is an illustration of a computer display and associated haptic boundaries. 
20 [001 3] Figure 3b is an illustration of the haptic space of a computer interface. 

[0014] Figure 3c is an illustration of a computer display with a separate control zone. 
[0015] Figure 3d is an illustration of a haptic space having an attendant control zone. 

Detailed Description of the Invention 

[0016] The present invention provides a method of human-computer interfacing that provides 
25 haptic feedback to control interactions such as scrolling or zooming within an application. Haptic 
feedback in the present method allows the user more intuitive control of interactions, and allows 
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# # 

the user's visual focus to remain on the application. The method comprises providing a control 
domain within which the user can control interactions. For example, a haptic boundary can be 
provided corresponding to scrollable or scalable portions of the application domain. The user can 
position a cursor near such a boundary, feeling its presence haptically (reducing the requirement 
5 for visual attention for control of scrolling of the display). The user can then apply force relative to 
the boundary, causing the interface to scroll the domain. The rate of scrolling or zooming can be 
related to the magnitude of applied force, providing the user with additional intuitive, non-visual 
control of scrolling. 

Haptic Control of Scrolling 

10 [0017] Figure 1 shows a computer display 103, including a portion thereof used to display a portion 
of an item such as a computer representation of a document 102. "Document" as used herein 
includes any information amenable to display with a computer, including without limitation text 
documents, pictures, drawings, word processing documents, and spreadsheets. Superimposed is 
an indication of a plane of a haptic space comprising the range of motion 104 of an input 

1 5 device (not shown). Note that, while the figure shows two dimensions for simplicity, the haptic 
space can comprise more than two dimensions. Even in a two dimensional application, a third 
dimension can be used, for example by making scrolling boundaries perceptible in front of (or 
nearer to) the user. A user can position a visual cursor 101 within the display by moving the input 
device. Movement of the input device can also determine the position of a haptic cursor, where the 

20 position of the haptic cursor is representative of the relation of the input device to haptic forces 
(e.g., the distance of the input device from a haptic boundary). The haptic cursor does not have to 
correspond to the visual cursor, but for ease of exposition we can consider the haptic cursor, 
within the input device range of motion 104, to correspond to the visual cursor, within the visual 
display 103. 

25 [0018] The portion of the item displayed 102 can be defined by visual 

boundaries 111, 112, 115, 116, defining ranges 121, 122. Visual boundaries can be part of the 
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display presented to the user, or can correspond to edges of the display itself, or can be not 
normally visible (as discussed later). The user can experience the document as though there were 
a window into the document. The user's view of the document is therefore limited by the 
boundaries of the window. The user can control the relative positioning of the window and 

5 document by controlling movement of the document, the window, or both. Relative motion in the 
plane of the document to expose various portions of the document provides an interaction 
commonly called scrolling. Relative motion normal to the plane of the document can control the 
apparent distance from the window to the document, changing the scale of the portion displayed 
(an interaction commonly called zooming). Both scrolling and zooming are contemplated by the 

10 term "interaction" as used herein. 

[0019] Conventional scrolling and zooming controls require the user to either explicitly control the 
interaction (e.g., by pushing arrow keys or moving a mouse over an object) or manipulate the 
visual cursor (e.g., by moving the visual cursor past the visual boundaries); manipulation of the 
visual cursor requires that the attention of the user be diverted from the document to the cursor 

15 control. The present invention can provide haptic boundaries to make interaction control more 
efficient. As with the haptic/visual cursors, the haptic boundaries do not have to correspond with 
the visual boundaries, but for ease of exposition we will consider them as corresponding. 
Consequently, in Figure 1, haptic boundaries are established within the range of motion 104 of the 
input device corresponding with the visual boundaries within the display. Movement of the visual 

20 cursor near a visual boundary accordingly corresponds with movement of the haptic cursor near 
the corresponding haptic boundary. When the haptic cursor approaches a haptic boundary, a force 
can be applied to the input device, resisting further motion of the haptic cursor relative to the haptic 
boundary (e.g., motion into or toward the haptic boundary). The user can thereby feel the 
boundaries of the portion of the item, providing greater control of the interaction (i.e., the rate of 

25 scrolling or zooming) and reducing the need for visual feedback. Once the user feels a boundary, 
the user can apply additional force to the input device, in opposition to the force applied by the 
interface, to control the interaction. The user accordingly can use the sense of touch (e.g., 
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kinesthetic and proprioceptive senses) to control the interaction, instead of visual, leaving the 
visual feedback path available for the remainder of the human-computer interface. The present 
invention also embraces similar interaction control by pulling (applying force away from the 
boundary) to control the interaction instead of pushing, and control of the interaction by forces 

5 applied by the user to a non-mobile force input device. 

[0020] The rate of scrolling can be determined from the force applied by the user. For example, if 
the user does not apply force greater than a threshold force, then the interface can determine that 
the user is just feeling the presence of the boundary and does not desire scrolling, and 
consequently can leave the visual display un-scrolled. Force in excess of the threshold can 

10 indicate that the user is pushing against the boundary of the window and desires that the window 
be moved (equivalent to scrolling, as discussed above), and the interface can change the display 
of the item to a portion adjacent the boundary. The scrolling rate can also be adjusted according to 
the magnitude of the force applied by the user. For example, greater applied force can initiate 
faster scrolling, providing an intuitive interface characteristic of "push harder to scroll faster". 

15 [0021] The method of the present invention is suitable for use in interfaces with various haptic or 
force feedback devices. It can be implemented in computer hardware or computer software, using 
conventional programming tools and computer systems. 

Haptic Control of Scrolling or Zooming Using a Control Zone 

[0022] Figure 2 shows a similar computer display arrangement as in Figure 1 , with the addition of 

20 a control zone 204. The control zone 204 is shown as overlaid visually on the display of the portion 
of the item; it can also be to a side, popped up when indicated, or not on the display at all. The 
control zone, when active, comprises all or a portion of the range of motion of the input device. In 
operation, the user indicates a transition into the control zone, then, once in the control zone, 
pushes against boundaries of the control zone to control scrolling or zooming of the item in the 

25 display. The desired interface characteristics can influence the accessibility of the control zone. 
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For example, the control zone can be accessible in a two-dimensional interface, at prescribed 
depths of a three-dimensional interface, or at all depths of a three-dimensional interface. 
[0023] The user can indicate a transition into the control zone in various ways. Several examples 
are presented below, 

5 One dimensional control zone. 

[0024] Figure 3a is an illustration of a computer display and associated haptic boundaries. The 
display comprises a control zone 331 near the right edge. The control zone 331 has haptic 
boundaries near its upper 332 and lower 333 extremes. The user can interact with the item (e.g., 
edit a document) when the cursor is not in the control zone. The user can move the cursor across 

10 the left boundary of the control zone, then push against either of the control boundaries 332, 333 
to haptically control scrolling. The transition into the control zone can be determined by the user, 
as examples, by visual feedback, or by applying a haptic resistive force that the user must 
overcome before allowing the cursor into the scrolling zone. Similar control zones can be provided 
for scrolling in other dimensions. The control zone itself can be left out of the visible display; the 

15 user then feels the transition to the control zone, then feels the control of the scrolling (freeing up 
the visual feedback path for attention to the item). 

[0025] An interface according to this example can be implemented according to the pseudocode 
below. 

1. Determine cursor .position; 

2. If {cursor .position e control . zone) then 

3. For all boundary in control . zone . boundaries 

a. If ( \ cursor .position - boundary \ < threshold) then 

i. Apply force directed away from boundary to input device; 
ii. Determine user . force directed toward boundary; 
iii . Scroll (direction {boundary) , rate ( user. force) ) ; 
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Three-dimensional motion into a control zone. 

[0026] Figure 3b is an illustration of the haptic space of a computer interface. A control zone 335 
allows control as discussed above: pushing against the haptic walls 336 from within the control 
zone controls scrolling in the corresponding direction. The haptic walls 336 extend from the base 

5 of the zone, so that, once in the zone, the user cannot easily slip out (the walls can also be inclined 
inward to make it even harder to accidentally slip out). The walls also discourage unintended 
scrolling. When the user moves a cursor near a haptic wall, the interface can apply a resistive 
force to discourage the cursor from traversing the wall. The user moving a cursor across the haptic 
space along a path that encounters the walls 336 will be prevented by the walls from entering the 

10 zone. Entry into the control zone requires explicit motion by the user to move over the walls (e.g., 
an explicit command or a three-dimensional cursor movement). 

[0027] The control zone can also comprise a closed portion of the space (e.g., a three dimensional 
box). The user can push through the walls to enter and exit, can supply an explicit command (e.g., 
voice or keyboard) for entry and exit, can touch a specific portion of the control zone (e.g., a switch 
15 inside the control zone) for entry or exit, or any combination. The use of a closed control zone 
allows a single control zone to be used to control interactions in all dimensions relevant to a 
document; e.g., scroll left and right using the sides of the control zone, scroll up and down using 
the top and bottom of the control zone, and zoom in and out using the front and back of the control 
zone. 

20 [0028] An interface according to this example can be implemented according to the pseudocode 
below. 

1. Determine cursor .position; 

2. If (cursor .position e control . zone) a 
(cursor .position < zone. boundary .height) then 
a. For' all boundary in control . zone. boundaries 

i. If (I cursor .position - boundary I < thresholdl) then 

1. Apply force directed away from boundary to input device; 

2. Determine user . force directed toward boundary; 

3. Scroll (user. force) ; 

3. If (cursor .position € control . zone) then 

a. If (I cursor .position - zone. boundary] < threshold2) a 



010-00-012; Anderson; Page 8 



(cursor .position < zone . boundary . height) then 

i. Apply force directed away from zone. bounds ryto input device 



Explicit transitions into a scrolling zone. 

[0029] Figure 3c is an illustration of a computer display with a separate control zone 341 . The 
control zone in this example can be made not accessible by simple cursor motion, eliminating the 
possibility of inadvertent cursor motion scrolling. The interface can provide for transition of the 

5 cursor into the control zone responsive to explicit direction from the user, for example, by voice 
command, keyboard control, or activation of a switch or button. Once in the control zone, the user 
can control scrolling by applying force against the boundaries of the zone, as discussed above. 
The interface can provide for transition out of the control zone by explicit direction, for example by 
the same mechanism as for entry, or by a simple inverse thereof. 

10 [0030] An interface according to this example can be implemented according to the pseudocode 
below. 

1. when (control .entry .command a not . in. control .mode) then 

a. cursor .position = initial, control . cursor .position 

b . in . control .mode 

2. when (control . exit . command a in . control .mode) then 

a. cursor .position = restore. cursor .position 

b. not . in . control .mode 

3. when (in . control .mode) 

a. determine cursor .position; 

b. For all boundary in control . zone . boundaries 

i. If (| cursor .position - boundary\ < threshold) then 

1. Apply force directed away from boundary to input device; 

2. Determine user . force directed toward boundary; 

3. Scroll (user. force) ; 



Attendant control zone. 

[0031] Figure 3d is an illustration of a haptic space having an always-attendant control 
zone 343, 344. The user can indicate a transition to control as discussed in the preceding 
15 example. The control zone 343 surrounds the cursor, but the zone's haptic boundaries are not 

active until the user indicates a transition to control. When the user indicates a transition to control, 
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the control zone 344 is made active surrounding the position of the cursor when the transition was 
indicated (adjusted, perhaps, when the cursor is too near some constraint such as an boundary of 
the computer space). The user has the control facility always available, and always in a consistent 
relationship to the cursor when called on. Once the control zone is activated, the user can control 

5 scrolling by applying force to the boundaries of the zone. The interface can provide for transition 
out of the control zone by a similar mechanism as used to enter, for example by a simple inverse 
(e.g., if pushing a button causes the zone to become active, then releasing the button can cause 
the zone to become inactive). The control zone accordingly need not consume any of the available 
display space when inactive, and, if made part of the haptic space but not the visual space, need 

10 never consume any of the available visual display space. An always-attendant control zone 
comprising a closed space can provide an intuitive interaction control: the control zone can be 
reliably accessed whenever needed, and all interaction controls provide the same intuitive haptic 
characteristics. 

[0032] An interface according to this example can be implemented according to the pseudocode 
1 5 below. 

1. when (control .entry . command a not . in . control .mode) then 
a . in . control .mode 

2. when (control . exit . command a in. control .mode) then 
a. not . in . control .mode 

3. when (in. control .mode) 

a. determine cursor .position; 

b. For all boundary in control . zone . boundaries 

i. If (| cursor '.position - boundary] < threshold) then 

1. Apply force directed away from boundary to input device; 

2. Determine user . force directed toward boundary; 

3. Scroll (user. force) ; 

[0033] The particular sizes and equipment discussed above are cited merely to illustrate particular 
embodiments of the invention. It is contemplated that the use of the invention may involve 
components having different sizes and characteristics. It is intended that the scope of the invention 
be defined by the claims appended hereto. 

20 
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